Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CMake Improvements #615

Merged

Conversation

stellar-aria
Copy link
Contributor

@stellar-aria stellar-aria commented Feb 22, 2024

Over the past year I've done some work on improving the state of the CMake build system, initially inspired by fixing building on native Windows, and then expanding in scope.

This is part 1 of the work towards fully migrating to CMake as per #606.

Changes in this PR include:

  • Fixing the native build on Windows
  • Adding support for LLVM/Clang
  • Adopting a similar build configuration process as the current Makefile (see example below)
  • Auto-detection of embedded build toolchains
  • Build the examples with the composite target examples (these are not part of the default all target)
  • Support for all the GoogleTest unit tests via CTest
  • Adding matrix toolchain testing to the Github Actions so we can test for regressions on older toolchains
  • Cleaning up the GitHub Actions a bit to use cached tools (for both GCC and LLVM). This makes running the test builds much faster.
  • Switch from using TOOLCHAIN_PREFIX to the CMake standard CMAKE_C_COMPILER for manually selecting a toolchain

Example project:

cmake_minimum_required(VERSION 3.26)
cmake_policy(SET CMP0048 NEW)

# Fetch libDaisy
include(FetchContent)
FetchContent_Declare(daisy
  GIT_REPOSITORY https://github.com/stellar-aria/libDaisy
  GIT_TAG cmake-only
)
FetchContent_MakeAvailable(daisy)

# Our project declaration

project(daisytest VERSION 0.0.1)

set(FIRMWARE_NAME daisytest)
set(FIRMWARE_SOURCES
  ${CMAKE_CURRENT_LIST_DIR}/src/test.cpp
)
#set(FIRMWARE_DEBUG_OPT_LEVEL -O0) # (optional) to customize Debug optimization level
#set(FIRMWARE_RELEASE_OPT_LEVEL -O2) # (optional) to customize Release optimization level

# DaisyProject uses FIRMWARE_NAME and FIRMWARE_SOURCES to build a target called ${FIRMWARE_NAME}
include(DaisyProject)

target_include_directories(${FIRMWARE_NAME} PUBLIC include)

Previously, builds would fail with a message stating that there's no such command "true".
CMAKE_C[XX]_ARCHIVE_FINISH is not a boolean option, but instead should be a string for how to run ranlib. This simply sets that to the default, but it could instead be removed in the future now that GCC binutils has lto automatically loading.
This removes the arguments that LLD cannot receive from the base default_build file, and moves them to the GCC-specific toolchain setup. This reformats all linker arguments to allow for CMake to auto-format them depending on the toolchain (clang uses -Xlinker instead of -Wl).
constexpr cannot take the result of a cast that is equivalent to a reinterpret_cast, as defined by the C++ standard. GCC may allow it, but it's technically illegal and clang errors.
Naked is not allowed for functions that contain bodies other than asm functions. Per the GCC docs: "The only statements that can be safely included in naked functions are asm statements that do not have operands. All other statements, including declarations of local variables, if statements, and so forth, should be avoided."

GCC silently allows this, but Clang notes this as a hard failure. Instead, the Default_Handler is marked as a generic interrupt, and the Reset_Handler is unmarked due to acting as the executable entrypoint.
This allows for configure-time selection of the linker script used by CMake, either at the command like with -DDAISY_STORAGE=qspi or in libraries that use libDaisy (by defining DAISY_STORAGE before inclusion).
Squashes 7+ warnings of this due to inclusion by other files.
This separates the massive root CMakeLists.txt file into libraries and places the respective configuration files closer to their source.
This ensures that we have architecture-specific configurations in a dedicated file while the LLVM and GNU toolchain files deal with their respective compilers only
config_ is never nullptr as it is initialized with the instantiation of a SaiHandle::Impl

This allows -Werror to work on GCC.
@stellar-aria
Copy link
Contributor Author

Yeah considering how heavily it's used in certain places of libDaisy, and even shown in the examples, I think that's a reasonable solution.

Ideally we could convert them to a static hang() function or something that's just while(1) { asm("nop"); } since that doesn't get utterly destroyed by the optimization passes, but that's a separate issue.

@stellar-aria
Copy link
Contributor Author

@beserge @stephenhensley any ideas when/how this might get merged? Are there any edits that need to happen for it to be done?

@beserge
Copy link
Contributor

beserge commented May 9, 2024

Checked out and built, everything appears as it should be. I'll defer to @stephenhensley on the merge timing, but looks good to go on my end! One thing to note, I had to update my CMake install, my out of date version was giving me some trouble, but once that was done it was smooth sailing.

@Segfault1602
Copy link

Any update on a merge timeline for this? This would be a very welcome addition!

@stellar-aria
Copy link
Contributor Author

Had to do manual merge resolution, since usbh_midi.c was already listed in the new refactored CMakeLists.txt. However, there also seems to be some issues with building on LLVM (on the CI side) now, so I'm going to poke about at that to see what's wrong.

@stellar-aria
Copy link
Contributor Author

Alright, everything's working as it should now.

Had to remove support for older versions of the LLVM Embedded Toolchain for ARM due to major breaking changes where the ARMv7E-M and ARMv7-M runtime libraries were merged, but considering we're only merging this now and 19.1.1 is the most current as of this comment, I think that's an acceptable solution.

@stephenhensley
Copy link
Collaborator

@stellar-aria Sounds good! I've been doing a little review/testing on this this week; I'll re-sync everything locally, and continue to test a bit more.

So far, so good. Since it's getting a bit late today, and I'd like to test converting a few projects with the latest version, I'll probably end up merging this on Monday unless any issues arise.

@stephenhensley
Copy link
Collaborator

So I am running into one issue that I didn't realize initially.
I've verified that I can get this to happen when building on Mac OS and Windows.

On Mac OS, I've built with ARM GCC 10.3-2021.10 (10.3.1 20210824 release)
On Windows I've built with ARM GCC 10-2020-q4-major (10.2.1 20201103 (release))

In both cases, programs built seem to run properly after being programmed to the Daisy.
After a reset, or a full power cycle, it looks like the program faults immediately from the Reset_Handler.

I've used the DaisyBlinkProject repo using the FetchContent feature, as well as modified the GPIO_Input example within the repo, and had the same experience.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.26)
cmake_policy(SET CMP0048 NEW)

include(FetchContent)
FetchContent_Declare(daisy
    GIT_REPOSITORY https://github.com/stellar-aria/libDaisy
    GIT_TAG 5086cc0
)
FetchContent_MakeAvailable(daisy)

project(daisyblink VERSION 0.0.1)
set(FIRMWARE_NAME daisyblink)
set(FIRMWARE_SOURCES
    ${CMAKE_CURRENT_LIST_DIR}/Blink/main.cpp
)
set(DAISY_GENERATE_BIN ON)

include(DaisyProject)

target_include_directories(${FIRMWARE_NAME} PRIVATE include)

main.cpp

#include "daisy_seed.h"

using namespace daisy;

DaisySeed hardware;

int main()
{
    hardware.Init();

    while (true)
    {
        hardware.SetLed(true);
        hardware.DelayMs(500);
        hardware.SetLed(false);
        hardware.DelayMs(500);
    }
}

Built with the following:

cmake -B build -G Ninja -D CMAKE_BUILD_TYPE=Release

cmake --build build --target all

I vaguely recall running into this a long time ago (possibly an issue with LTO, or something), but it was several years ago.

I'll retry with different optimization, etc. and see if I can narrow down the issue, but it's also possible I'm just overlooking something obvious.

@stellar-aria
Copy link
Contributor Author

@stephenhensley I've used your test case and it compiles and works (uploaded via the Daisy web programmer) with GCC 13.2.1 (see attached files) even after a cold-boot and power cycle, so next I'm going to try with 10-2020-q4.
daisyblink.zip

@stellar-aria
Copy link
Contributor Author

I can confirm the behavior on 10-2020-q4, I'm going to check versions upwards to see where it resolves

@stellar-aria
Copy link
Contributor Author

Issue is present in 11.3.1 and resolved in 12.2.1

@stephenhensley
Copy link
Collaborator

@stellar-aria thanks for stepping through that, and finding where it improved! -- I'll continue testing with a newer compiler.

Aside from this issue everything seems great so far. Last things I want to check are bootloader/linker script related, but I don't anticipate any issues.

@stellar-aria
Copy link
Contributor Author

stellar-aria commented Oct 14, 2024

yeah I'm not sure where this is going wrong. It seems like there might be a problem with linker stuff due to it just failing in the reset handler. For fun, here's a comparison between 11.3 and 12.2's first few bytes of the bin file. Unfortunately, I've never been good at decoding hex by hand.
image

edit: it might be some kind of alignment issue?

@asavartsov
Copy link
Contributor

asavartsov commented Oct 15, 2024

TLDR: Remove -fno-builtin from stm32h750xx.cmake.

ARM GCC 10-2020-q4-major with example above in Release configuration (-O3) generates floating point instructions (vldr, vstmia) for the memory copy loop (😮) in the Reset_Hanlder() that is running before FPU enabled at SystemInit(). It causes a NOCP hard fault.

image

GCC will usually optimize this loop to builtin memset but CMake build has -fno-builtin flag set (which usage is strongly discouraged anyway). This flag is not set in any of Makefiles, this is why it always worked before.

In my test with real HW removing -fno-builtin from stm32h750xx.cmake fixes the issue.

@stellar-aria
Copy link
Contributor Author

Thank you so much for figuring that out @asavartsov. It looks like that was added in one of the first few refactor commits I was doing, probably pulled from somewhere else originally, and it's been so long I have no idea what prompted it being there or what I was thinking 😅

It looks like without it, there's now some size changes in some of the binaries for clang, which is preventing them from fully building, so some of the examples might have to be prodded to work properly or removed from the test build

@stellar-aria
Copy link
Contributor Author

The culprit seems to be that without -fno-builtin, clang is placing some structures in the .data section that were previously in .bss, specifically the global variables in the main file (hw, midi, event_log).
image

@stephenhensley
Copy link
Collaborator

The culprit seems to be that without -fno-builtin, clang is placing some structures in the .data section that were previously in .bss, specifically the global variables in the main file (hw, midi, event_log).

Very strange indeed. Good catch.

@asavartsov thanks for digging in, and clarifying what was going on a bit more!


I think for the time being, we can except any of the currently included examples that are too large in order to get an initial version of this merged/passing CI again.

Another, slightly more medium-term, solution would be to build those examples with the bootloader (minding that at least the SDMMC example would require a custom linker so the FIL objects, etc. can be in AXI SRAM). However, I think coverage for all examples w/ CMake can be a goal we work toward rather than requiring it as a part of this PR.

I haven't personally run into any further issues yet, but I haven't had a ton more time for further testing. This also doesn't really have much effect on the (current) primary Make-based build system that seems unaffected by any of the llvm/cmake build issues we've discussed in the last few days.

Let me know what you think, and we can go from there.

@stellar-aria
Copy link
Contributor Author

@stephenhensley Yeah, I'm in agreement. Investigation on my end has indicated that this behavior is actually Clang/LLVM more strictly conforming to the C++ standard than GCC does, which moves this from a codebase error to something more complicated. I might poke some LLVM devs and ask if they have any suggestions for how to go about achieving the behavior we want (large array-like globals in .bss).

In the meantime, I've reduced the Clang CI's build tasks from all of the examples down to just BootloaderBlink, which should at least verify that the CI works, even if not for all of our general GCC test cases. Meanwhile the GCC stuff is all up to date with the current collection of examples.

I'd say this is probably good to merge at this point, as long as you find no other issues. Bringing up conformance between the two compilers can definitely be another PR, if at all.

@asavartsov
Copy link
Contributor

asavartsov commented Oct 17, 2024

In this particular case one could (and probably should) put MIDII/FIFO objects (which will take buffers with them) in any other section of their choice outside of FLASH region manually. Or avoid globals and trust compiler but it would be a bit trickier if you need to interact with audio callback since you can’t pass anything (I would imagine at least some opaque pointer) to it. So means to override this more or less exist for end user.

@stephenhensley
Copy link
Collaborator

Alright! I'm signing off.
Agreed that the CI validation is a good initial step, and full build coverage can be worked on over time.

Thanks again, @stellar-aria for the tremendous PR! And sorry for how long it took to get it merged.
I'm trying to spend a little bit of time on the library each week now. So things should move a little more consistently.
This was one of the last things to get in (aside from some QSPI/bootloader stuff I'll squeeze in next week) before the we do a v8 release with all of the recent merges.

@stephenhensley stephenhensley merged commit bd13385 into electro-smith:master Oct 18, 2024
13 checks passed
@stellar-aria stellar-aria deleted the feature/cmake-improvements branch October 19, 2024 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants